Method and apparatus for thumbnail generation for a video device

ABSTRACT

Systems and processes are provided to generate and transmit a thumbnail image for a video searching function including a video input configured to receive a video signal, a processor configured to decode the video signal to generate a video program, to generate a plurality of thumbnail images in response to the video program, to determine a subset of the plurality of thumbnail images in response to a client request and to couple the subset of the plurality of thumbnail images to a network interface, and the network interface configured to transmit and receive data on a local area network, the network interface being operative to receive the client request from a client device and to transmit the subset of the plurality of thumbnail images to the network interface for transmission to the client device.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No. 16/722,089, filed Dec. 20, 2019, which is incorporated herein by reference.

TECHNICAL FIELD

The following discussion generally relates to the decoding of video on an electronic device and generation of preview image thumbnails. More particularly, the following discussion relates to minimizing a memory requirement for a video decoder and generating a plurality of low resolution images in response to the decoded video.

BACKGROUND

Digital video programming is transmitted in an encoded format according to standardized formats, such as Advanced Television Systems Committee (ATSC) or Moving Picture Experts Group Four (MPEG-4), to compress the video data in order to reduce the amount of data required to be transmitted to transmit an audio/video signal. Decoding the video at a user device, such as a digital television or digital set-top box, typically requires a video decoder processor. As video increases in resolution, the decoding process requires increased processing resources and memory to decode the video. This results in a problem for legacy devices which employ earlier video decoders, slower processing speeds and limited memory for performing the decoding process.

When a user of a digital video device wishes to fast forward or rewind the video, typically thumbnail images are displayed so that a user may know the current location in the video during the operation. The thumbnails are generated from the decoded video. Typically, digital video devices are only equipped with one video decoder and there is not a special thumbnail decoder available. Thus, decoding the video for thumbnails may double the processing requirements and/or memory requires as the video must be decoded as a normal high definition video/ This presents a problem, especially on lower powered legacy devices, when a user wishes to advance or rewind a program and the digital video device unable to generate preview thumbnails to help the viewer to locate a desired location within the program due to a lack of resources. It is therefore desirable to create systems and methods to provide preview thumbnails on a digital video device to avoid these problems. Other desirable features and characteristics will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and this background section.

BRIEF SUMMARY

Disclosed herein are human machine interface methods and systems and related circuitry for provisioning computational systems, methods for making and methods for operating such systems, and system equipped with onboard control systems. By way of example, and not limitation, presented are various embodiments of thumbnail generation for video location preview in a video device.

In accordance with an aspect of the disclosure

The above advantage and other advantages and features of the present disclosure will be apparent from the following detailed description of the preferred embodiments when taken in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

The exemplary embodiments will hereinafter be described in conjunction with the following drawing figures, wherein like numerals denote like elements, and wherein:

FIG. 1 shows an environment for thumbnail generation for a video device according to an exemplary embodiment of the present disclosure.

FIG. 2 shows a block diagram illustrating a system for thumbnail generation for a video device according to an exemplary embodiment of the present disclosure.

FIG. 3 shows a flowchart illustrating a method for thumbnail generation for a video device according to an exemplary embodiment of the present disclosure.

FIG. 4 shows a flowchart illustrating a method for providing thumbnail images in response to a request from a client device according to another exemplary embodiment of the present disclosure

The exemplifications set out herein illustrate preferred embodiments of the disclosure, and such exemplifications are not to be construed as limiting the scope of the disclosure in any manner.

DETAILED DESCRIPTION

The following detailed description is intended to provide various examples, but it is not intended to limit the disclosure or the application and uses of the disclosure. Furthermore, there is no intention to be bound by any theory presented in the preceding background or the following detailed description.

The general concepts described herein may be implemented in any connected device that may be used for video streaming or the like. For convenience of illustration, the following discussion often refers to a video signal processing device operative to generate video and/or video thumbnail information for presentation on a video display or the like. The actions taken in response to the generation of set top box video thumbnails may also be associated with the functionality of the mobile phones, portable electronic entertainment devices, and any other electronic media device capable of image or video display and/or playback. Additional details and examples are provided in the following description.

Turning now to FIG. 1 , an environment for a system 100 for thumbnail generation in a video device according to an exemplary embodiment of the present disclosure is shown. The exemplary system 100 may include a display 110, a video set top box (STB) 120, and an antenna 130. According to an exemplary embodiment, the display 110 may be operative to display a video program 111, a video search bar 113 and one or more thumbnail 112 images representative or images within the video program.

In this exemplary embodiment, the STB 120 is operative to receive a video program via the antenna 130. Alternatively, the STB 120 may receive the video program via a transmission cable, such as a coaxial cable, a fiber optic cable, a wireless network connection or the like. The STB 120 may be operative to store the video program in an electronic format on a hard disk drive HDD or the like. The STB 120 may be operative to convert the video program from a first format, such as ATSC to a second format, such as MPEG-4 for program storage on the HDD.

The STB 120 may be further operative to generate a thumbnail file having a plurality of periodically timed images representative of locations within the video program. For example, the STB 120 may be operative to decode the stored video program, save a frame of the video program for every five seconds of video program run time. The frame may be stored as a JPEG image on the hard drive with a metafile storing information as to the name, location, video program indicator and time within the video program of the image. For example, for a one minute video, the STB 120 may be operative to store 12 images representative of a frame of the video every five seconds.

The STB 120 may advance the video program or rewind the video program in response to a request by a user. The user may be operative to generate the request in response to a button push on a remote control, a voice command, or the like. The STB 120 may then be operative to generate a user interface or an overlay over the image 111 to be displayed before coupling to the display 110. The user interface may include a progress bar 113 to display a current time location within the video program. When displaying the progress bar 113, the client 120 may be further operative to display a series of thumbnails 112 where the thumbnails are images representing frames in the video program situated at time locations before and after the current time location.

Turning now to FIG. 2 , a system 200 for thumbnail generation for a video device according to an exemplary embodiment of the present disclosure is shown. The system 200 may include a video input 205, a video decoder 210, a processor 220, a network interface 230 and a memory 240. The video input 205 is operative to receive a video program and/or audio-visual program. The video input 205 may include a radio frequency or microwave tuner operative to receive a data representative of the video program modulated on a radio frequency or microwave carrier signal. Alternatively, the video input 205 may be a network interface operative to receive data formatted for network transport, such as TCP/IP and to extract the video program from the transport packets.

The video input 205 may be a cable television tuner, or the like, operative to receive an ATSC, NTSC, or quadrature amplitude modulated (QAM) signal for transporting the video program. In this exemplary embodiment, the signal may be received via a cable television network, or satellite television network, or antenna based network.

The video decoder 210 may be operative to receive the signal from the video input 205 and to decode the signal to generate a video signal in a format suitable for the processor 220. The video decoder 210 may be operative to demodulate the signal from the video input to extract the video program and to couple the video program to the processor 220. In an exemplary embodiment, the video decoder 210 may be further operative to couple the video signal to a video output for display on a video display or the like.

The processor 220 is operative to receive a video signal from the video decoder 210 and to format the video signal in a format suitable for storage in the memory 240. For example, the processor 220 may receive the video signal from the video decoder 210, encode the video signal using video compression techniques into a format such as H.264 or MPEG-4, and then store the compressed video file onto the memory 240. The processor 220 may be further operative to retrieve the compressed video file in response to a user request and to couple the video signal to a display or the like. In addition, the processor 220 may be operative to couple the video file to the network interface 230 for transmission to a client device in response to a request from the client device. In an exemplary embodiment, the processor 220 may control streaming of the video file in a video streaming format for coupling to the client device via the network interface 230.

In addition, the processor 220 is operative to generate a thumbnail file for the compressed video file and to store the thumbnail file on the memory 240. To generate the preview thumbnails of the video program, the processor 220 is operative to read the first frame of the stored video program and to determine if the frame is an I-frame, also known as an intra-frame. If the frame is an I-frame, or a complete image frame, the processor 220 then determines if the elapsed video time has exceeded a minimum duration. A thumbnail may be generated as a predetermined minimum interval in order to reduce the number of thumbnails being generated. If the frame is an I-frame and the time since generating the previous thumbnail is greater than the threshold interval, the processor 220 then decodes the frame and encodes the image as a compressed image, such as a JPEG. The processor 220 may further be operative to resize the image and/or decrease the image resolution to reduce the image size for storage and transmission. The processor 220 is then operative to store the image on the memory.

The processor 220 may further be operative to generate or append a metafile indicative of the set of thumbnail images generated for the video program. The data stored in the metafile may include frame pts, byte offset which may be the byte offset of the thumbnail in the mtn file, thumbnail file size, the frame or file offset in the file, thumbnail file name, and/or timestamp in milliseconds.

The processor 220 is further operative to receive a request from a client device via the network interface 230 requesting a subset of the thumbnails in response to a video search operation, such as rewinding or fast forwarding the video. The request may indicate the video program being viewed, the location or time progress within the video program and a direction and speed of the search operation being performed. The processor 220 is then operative to consider the speed of the search operation to determine a suitable time interval between thumbnails. For example, if the user is searching the video program at a high speed, thumbnails every one minute may be appropriate wherein a user searching at a slower speed may require thumbnails every five seconds.

The processor 220 may then couple the determined thumbnails corresponding to the video program, time progress, and speed and couple these thumbnail images to the network interface 230 for transmission to the client. The processor 220 may further generate a metafile indicative of the thumbnail images and information related to the thumbnail images and couple this metafile to the network interface 230 for transmission to the client.

The network interface 230 is operative to transmit and receive data from client devices on a network. For example, the network interface 230 may be part of a DVR and is operative to receive requests from one or more client devices on a network. The network interface 230 is then operative to transmit a list of available programs, a requested program, a thumbnail data file, or other information to a client device in response to a request from the client device. The network interface 230 may communicate directly with the client device or may communicate via a network router on a wireless local area network, such as a Wi-Fi network.

Turning now to FIG. 3 , a method 300 for thumbnail generation for a video device according to an exemplary embodiment of the present disclosure is shown. The method is first operative to receive 305 a video program via a network interface, video input or the like. The video program may be transmitted via a radio frequency or microwave frequency carrier signal and transmitted via wireless or wired transmission means. In one exemplary embodiment, the video program may be received via a satellite transmission and received at a parabolic antenna equipped with a low noise block amplifier and downconverter.

The method may be next operative to store 310 the video program on a storage medium within, or accessible to, the DVR. The video program may be demodulated and/or decoded by a demodulator or video decoder, before being stored in the memory. The video program may be stored in an H.264, MPEG-2, MPEG-2, or other video file compression format.

To generate the thumbnails according to an exemplary embodiment of the present disclosure, the method is next operative to read 315 the next frame of the video program stored in the memory to establish the suitability of the frame as a source of a thumbnail. To establish the suitability, the method is first operative to determine 320 if the next frame is an I-frame. An I-frame, or intra-frame, is a complete image that does not require adjacent frames for decoding. If the frame is not an I-frame, the method is operative to return to read 315 next frame of the video program.

If the frame is determined 320 to be an I frame, the method is next operative to determine 325 if an elapsed time between frames is greater than elapsed time threshold, where the elapsed time threshold is a minimum time between thumbnails in order to limit the number of thumbnails being generated and the memory required to store the thumbnails. If the elapsed time between frames is not greater than elapsed time threshold, the method is operative return to read 315 next frame in the video program. If elapsed time between frames is greater than elapsed time, the method is next operative to decode 330 the frame and to encode 335 the frame as an image. The method may resize the image and/or decrease the image resolution to reduce the image size for storage and transmission.

Once the frame is encoded as an image in the suitable format, the method is then operative to write 340 image to a hard disk drive or other storage medium. In one exemplary embodiment, the method may be operative to generate a multi-thumbnail file (MTN) and to append the image to the MTN. As an alternative to storing the image as an individual thumbnail, the use of the concatenated MTN reduces memory requirements on the hard disk drive to block alignment and the like. The method is then operative to append 345 metadata to image file and to return to read 315 next fame of the video program.

In an exemplary embodiment, caching logic may be employed to improve the thumbnail experience by reducing the thumbnail load speed for thumbnails stored on the hard disk drive. This caching logic may be performed in the random-access memory of a device requesting the thumbnails. For longer video programs, such as three-hour movies or sporting events, it is not practical to transmit all the thumbnails in the program as the requesting device may be limited on available storage resources, such as random-access memory. To reduce the number of thumbnails transmitted and cached, the method may be operative to transmit a subset of the thumbnails based in the current program viewing time position. For example, if a user of the requesting device is watching 4-hour video program and is at 10-minute mark of the video program, the method may only transmit thumbnails from 5 to 20 minutes of the video program. Alternatively, a larger number of the thumbnails may be transmitted, and the caching logic of the requesting device may perform the selective caching. For example, the method may further employ a MTN request handling routine to handle the thumbnail requests from requesting devices. In an exemplary embodiment, the requesting device may request a number of thumbnails based on the program viewing location and the method may be then operative to transmit an MTN file with meta information on details of each thumbnail file.

Turning now to FIG. 4 , a method 400 for providing thumbnail images in response to a request from a client device according to an exemplary embodiment of the present disclosure is shown. The method is operative to receive a request for a set of thumbnails relating a video program from a client device via a network connection or the like, and to provide a subset of the stored thumbnails related to the video program in response to the request. The method may be performed by a DVR or other network video storage device operative to stream video to a client device.

In this exemplary embodiment, the method is first operative to transmit 410 a video program to a client device. The method may stream, or continuously transmit a sequential data stream of the video program such that the receiving device does not need to download the complete video program to display the video program. The client device may be operative to receive packets of data related to the video program, to buffer and order the received packets into a correct sequence, and then sequentially display the buffered video data as more video data is being received, ordered and buffered. Buffering the video data compensates for delays in the transmission process and lost/retransmitted packets without interrupting the presentation of the video program on the display.

The method is next operative to receive 420 a request for thumbnail images from the client device. When the user performs a seek operation on a client device while watching a program being streamed from a DVR or the like, and the progress bar on the client device is activated in response to a video search request by the user. The video search request may be a fast forward or rewind operation. The request for thumbnail images from the client device may include a timestamp of the location in the video program that the progress bar was activated. The request may further include a speed of the active video search. For example, if a high-speed forward search is activated or a slow forward search is activated the speed at which the search is being performed may be indicated in the request.

The method is next operative to determine 430 a subset of thumbnails in response to the request. For example, in response to the program being streamed to the client device, the timestamp of the location in the video program, and the speed of the search, the method may be operative to determine that thumbnails spaced every 10 seconds for a one minute period before and after the timestamp for the video program should be transmitted. The method may be operative to generate a subset metafile, such as a json file, listing the subset of thumbnail names, times, file size, etc. A thumbnail offset, in bytes, may also be included in the metafile to be used for seeking by the client device in order to aid in accuracy. The method is then operative to transmit 440 the subset of thumbnails and the subset metafile to the client device and return to transmitting the video program.

The term “exemplary” is used herein to represent one example, instance or illustration that may have any number of alternates. Any implementation described herein as “exemplary” should not necessarily be construed as preferred or advantageous over other implementations. While several exemplary embodiments have been presented in the foregoing detailed description, it should be appreciated that a vast number of alternate but equivalent variations exist, and the examples presented herein are not intended to limit the scope, applicability, or configuration of the disclosure in any way. To the contrary, various changes may be made in the function and arrangement of the various features described herein without departing from the scope of the claims and their legal equivalents. 

The invention claimed is:
 1. A method performed by a video streaming device comprising: receiving a video program by the video streaming device; storing the video program on a storage medium; generating, by the video streaming device, a plurality of thumbnail images in response to the video program; generating, by the video streaming device, a first metafile associated with the plurality of thumbnail images wherein the first metafile includes an indicator for each of the plurality of thumbnail images; streaming, by the video streaming device, the video program to a client device via a network; receiving, by the video streaming device via the network, a request from the client device indicative of a search operation wherein the request includes a timestamp associated with the video program and a search speed; determining by the video streaming device, a subset of the plurality of thumbnail images in response to the request wherein the subset of the plurality of thumbnail images is determined in response to the timestamp associated with the video program and the search speed; and generating, by the video streaming device, a second metafile indicative of the subset of the plurality of thumbnail images wherein the second metafile includes an indicator for each of the subset of the plurality of thumbnail images; and transmitting, by the video streaming device, the subset of the plurality of thumbnail images and the second metafile to the client device via the network.
 2. The method of claim 1 wherein the plurality of thumbnail images are interframe images.
 3. The method of claim 1 wherein the indicator for each of the plurality of thumbnail images includes a name and time indicator for each of the plurality of thumbnail images.
 4. The method of claim 1 wherein the indicator for each of the plurality of thumbnail images includes at least one of a name and time indicator for each of the plurality of thumbnail images.
 5. The method of claim 1 wherein the storage medium is a hard disk drive.
 6. The method of claim 1 wherein the indicator for each of the plurality of subset of the plurality of thumbnail images includes a name and a time indicator.
 7. The method of claim 1 wherein the indicator for each of the plurality of subset of the plurality of thumbnail images includes at least one of a name and a time indicator.
 8. The method of claim 1 wherein each of the plurality of thumbnail images is generated in response to an interframe of the video program and a time interval between interframes of the video program.
 9. An apparatus comprising: a video input configured to receive a video signal; a processor configured to decode the video signal to generate a video program, to generate a plurality of thumbnail images in response to the video program, to generate a first metadata file having an indicator for each of the plurality of thumbnail images, to determine a subset of the plurality of thumbnail images in response to a client request, wherein the client request includes a timestamp associated with the video program a search speed, to generate a second metadata file having an indicator for each of the subset of the plurality of thumbnail images in response to the timestamp associated with the video program and the search speed, and to couple the subset of the plurality of thumbnail images to a network interface; and the network interface configured to transmit and receive data on a local area network, the network interface being operative to receive the client request from a client device and to transmit the subset of the plurality of thumbnail images and the second metadata file to the network interface for transmission to the client device.
 10. The apparatus of claim 9 further including a memory for storing the video signal and the plurality of thumbnail images.
 11. The apparatus of claim 9 wherein the indicator for each of the plurality of thumbnail images includes a name and time indicator for each of the plurality of thumbnail images.
 12. The apparatus of claim 9 wherein the indicator for each of the plurality of thumbnail images includes at least one of a name and time indicator for each of the plurality of thumbnail images.
 13. The apparatus of claim 9 further including a demodulator for demodulating the video signal to generate a demodulated video signal and wherein the video program is generated in response to the demodulated video signal.
 14. The apparatus of claim 9 wherein the indicator for each of the plurality of subset of the plurality of thumbnail images includes a name and a time indicator.
 15. The apparatus of claim 9 wherein the client request is indicative of a time location in the video program.
 16. The apparatus of claim 9 wherein the client request is indicative of a time location in the video program, a time duration between each of the subset of the plurality of thumbnail images is determined in response to the search speed.
 17. The apparatus of claim 9 wherein each of the plurality of thumbnail images is generated in response to an interframe of the video program.
 18. The apparatus of claim 9 wherein each of the plurality of thumbnail images is generated in response to an interframe of the video program and a time interval to a previous thumbnail image.
 19. An apparatus for generating and transmitting a thumbnail image comprising: a tuner for receiving and demodulating a video signal to generate a video program; a memory for storing the video program; a processor for generating a plurality of thumbnail images in response to the video program, wherein each of the plurality of thumbnail images are separated by a duration of time within the video program, for generating a first metadata file having an indicator for each of the plurality of thumbnail images, for receiving a client request including a timestamp associated with the video program and a search speed, the processor being further operative to determine a subset of the plurality of thumbnail images in response to the timestamp associated with the video program and the search speed, to generate a second metadata file having an indicator for each of the subset of the plurality of thumbnail images and to couple the subset of the plurality of thumbnail images and the second metadata file to a network interface; and a network interface for receiving the request from a client device and to receive the subset of the plurality of thumbnail images and the second metadata file from the processor and to transmit the subset of the plurality of thumbnail images and the second metadata file to the client device.
 20. The apparatus for generating and transmitting a thumbnail image of claim 19 wherein the indicator for each of the plurality of thumbnail images includes at least one of a name and time indicator for each of the plurality of thumbnail images. 